Papers with long-context understanding
AutoFigure-Edit: Generating Editable Scientific Illustrations via Reference-Guided Styling (2026.acl-demo)
Copied to clipboard
Zhen Lin, Qiujie Xie, Minjun Zhu, Shichen Li, QiYao Sun, Enhao Gu, Yiran Ding, Ke Sun, Fang Guo, Panzhong Lu, Zhiyuan Ning, Yixuan Weng, Yue Zhang
| Challenge: | Existing automated systems for scientific illustrations are limited in editability, stylistic controllability, and efficiency. |
| Approach: | They propose an end-to-end system that generates fully editable scientific illustrations from long-form scientific text while enabling flexible style adaptation through user-provided reference images. |
| Outcome: | The proposed system generates fully editable scientific illustrations from long-form scientific texts while enabling flexible style adaptation through user-provided reference images. |
Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problems (2026.tacl-1)
Copied to clipboard
| Challenge: | Existing approaches to improve long-context understanding of large language models lack scalability and stability. |
| Approach: | They challenge the assumption that SC’s benefits generalize to long-context settings . they find that persistent position bias degrades performance on long-consistency tasks . |
| Outcome: | The proposed approach fails to improve and actively degrades performance on long-context tasks. |
RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios (2025.acl-long)
Copied to clipboard
| Challenge: | RuleArena assesses the ability of large language models (LLMs) to follow complex, real-world rules in reasoning. |
| Approach: | They propose a benchmark to evaluate the ability of large language models (LLMs) to follow complex, real-world rules in reasoning. |
| Outcome: | The proposed benchmark covers airline baggage fees, NBA transactions, and tax regulations. |
Extending LLM Context Window with Adaptive Grouped Positional Encoding: A Training-Free Method (2025.acl-long)
Copied to clipboard
| Challenge: | Existing long-context training data is scarce and requires substantial GPU resources for training. |
| Approach: | They propose a training-free plug-and-play method to enhance long-context understanding in existing large language models. |
| Outcome: | The proposed method outperforms existing LLMs on various tasks and surpasses baseline methods. |
S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Model (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks fail to evaluate extremely long-context LLMs or analyze their limitations. |
| Approach: | They propose a Synthetic, Scalable, Systematic evaluation suite for LLMs using SQL execution. |
| Outcome: | The proposed evaluation suite is able to scale text length and difficulty across scenarios and provides strong correlations with real-world benchmarks. |
Token Weighting for Long-Range Language Modeling (2025.findings-naacl)
Copied to clipboard
| Challenge: | Many applications of large language models (LLMs) require long-context understanding, but models still struggle with such tasks. |
| Approach: | They propose token-weighting schemes that assign different weights to each training token in the loss, generalizing existing works. |
| Outcome: | The proposed methods compare confidences of a long-context and short-concept model and show that non-uniform loss weights improve the long-constability of LLMs. |
Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing long-text evaluation benchmarks, such as L-Eval and LongBench, focus on QA and summarization tasks. |
| Approach: | They propose a length-adaptable benchmark for evaluating the long-context understanding of large language models. |
| Outcome: | The proposed benchmarks do not cover ultralong settings (100k+ tokens) and are difficult to evaluate across different length ranges. |
BRIEF: Bridging Retrieval and Inference for Multi-hop Reasoning via Compression (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to augment language models with external knowledge but they are limited by static nature of pre-training data. |
| Approach: | They propose a lightweight approach that compresses retrieved documents into highly dense textual summaries to integrate into in-context RAG. |
| Outcome: | The proposed approach reduces latency and costs while achieving high performance in open-domain questions. |
Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA (2024.emnlp-main)
Copied to clipboard
Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, Yongbin Li
| Challenge: | Existing benchmarks for evaluating long-context language models employ irrelevant noise texts to artificially extend the length of test cases, diverging from the real-world scenarios of long-constituency applications. |
| Approach: | They propose a long-context benchmark, Loong, aligning with realistic scenarios through extended multi-document question answering (QA) . |
| Outcome: | The proposed model can scale up the context window of large language models to perform in-depth analysis of multiple long documents. |
Facilitating Long Context Understanding via Supervised Chain-of-Thought Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have enabled them to process increasingly longer sequences, ranging from 2K to 2M tokens and even beyond. |
| Approach: | They propose a synthetic dataset in the financial domain that integrates Chain-of-Thought reasoning into LLMs in a supervised manner to facilitate effective long-context understanding. |
| Outcome: | The proposed model outperforms standard GPT-4o-mini on the Loong benchmark and fine tunes LLaMA-3.1-8B-Instruct on the model, achieving a 28.0% gain on the financial subset. |
LC-Eval: A Bilingual Multi-Task Evaluation Benchmark for Long-Context Understanding (2025.findings-emnlp)
Copied to clipboard
Sheikh Jubair, Arwa Omayrah, Amal Alshammari, Alhanoof Althnian, Abdulhamed Alothaimen, Norah A. Alzahrani, Shahad D. Alzaidi, Nora Al-Twairesh, Abdulmohsen Al-Thubaity
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated sophisticated capabilities, including the ability to process and comprehend extended contexts. |
| Approach: | They propose a bilingual, multi-task evaluation benchmark designed to evaluate long-context understanding in English and Arabic. |
| Outcome: | The proposed benchmark targets context lengths ranging from 4k to over 128k tokens. |
VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization (2026.acl-long)
Copied to clipboard
| Challenge: | Existing quantization methods for large language models suffer performance degradation at ultra-low bit-widths due to key cache outliers. |
| Approach: | They propose a vector quantization method that suppresses outliers in the key cache and reduces memory access overhead. |
| Outcome: | The proposed method outperforms baseline quantization methods across long-context understanding and mathematical reasoning tasks while minimizing memory access overhead. |
LiveLongBench: Tackling Long-Context Understanding for Spoken Texts from Live Streams (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that spoken text exhibits unique linguistic properties, such as high redundancy and repetitive phrases. |
| Approach: | They propose a long-text dataset that better handles redundancy in spoken text . their results highlight key limitations of current methods and suggest future directions . |
| Outcome: | The proposed benchmark improves existing methods and improves on redundancy in spoken text. |
DRPG (Decompose, Retrieve, Plan, Generate): An Agentic Framework for Academic Rebuttal (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to support academic rebuttal rely on off-the-shelf LLMs or simple pipelines that struggle with long-context understanding. |
| Approach: | They propose an agentic framework for automatic academic rebuttal generation that operates through four steps: Decompose reviews into atomic concerns, Retrieve relevant evidence from the paper, Plan refortations, and Generate responses accordingly. |
| Outcome: | The proposed framework outperforms existing rebuttal pipelines and achieves 98% accuracy beyond the average human level using only an 8B model. |